Multi-threaded processing with reduced context switching

ABSTRACT

Multi-threaded processing with reduced context switching is disclosed. Context switches may be avoided through the use of pre-emption notification, a pre-emption wait time attribute and a no-context-save yield.

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application Ser. No.11/839,492, to John P. Bates, filed Aug. 15, 2007, the entire contentsof which are incorporated herein by reference.

FIELD OF THE INVENTION

This invention generally relates to parallel processing and moreparticularly to multi-threaded operation in parallel processors such ascell processors.

BACKGROUND OF THE INVENTION

A major advance in electronic computation has been the development ofsystems that can perform multiple operations simultaneously. Suchsystems are said to perform parallel processing. One type of parallelprocessing system, known as a Cell processor, has been developed toimplement parallel processing on electronic devices ranging fromhandheld game devices to main frame computers. A typical Cell processorhas a main memory, power processor element (PPE) and up to 8 additionalprocessor elements referred to as synergistic processing elements (SPE).Each SPE is typically a single chip or part of a single chip containinga processor known as a synergistic processor unit (SPU) and a localmemory. All of the SPEs and the PPE can access the main memory, e.g.,through a memory flow controller (MFC). The SPEs can perform parallelprocessing of operations in conjunction with a program miming on thePPE. The local memory associated with each SPU is relatively small,currently about 256 kilobytes in one common implementation. This memorymust be managed by software to transfer code and data to/from the localSPE memories.

The SPE have a number of advantages in parallel processing applications.For example, the SPE are independent processors that can execute codewith minimal involvement from the PPE. Each SPE has a high direct memoryaccess (DMA) bandwidth to RAM. An SPE can typically access the mainmemory faster than the PPE. In addition each SPE has relatively fastaccess to its associated local store. The SPE also have limitations thatcan make it difficult to optimize SPE processing. For example, the SPEhave no coherent memory and no hardware cache. In addition, many commonprogramming models do not work well on SPE.

A typical SPE process involves retrieving code and/or data from the mainmemory, executing the code with the SPU to manipulate the data, andoutputting results of the manipulation of the data to main memory or, insome cases, another SPU. To achieve high SPU performance it is desirableto optimize the above SPU process in relatively complex processingapplications. For example, in applications such as computer graphicsprocessing SPUs typically execute tasks thousands of times per frame.

One prior art task management system used for Cell processors and othertypes of processors is based on a software concept referred to as“threads”. A “thread” generally refers to a part of a program that canexecute independently of other parts. Operating systems that supportmultithreading enable programmers to design programs whose threadedparts can execute concurrently. When a thread is interrupted, a contextswitch may swap out the contents of an SPE's local storage to the mainmemory and substitute 256 kilobytes of data and/or code into the localstorage from the main memory where the substitute data and code areprocessed by the SPU. A context switch is the computing process ofstoring and restoring the state of a SPE or PPE (the context) such thatmultiple processes can share a single resource.

A typical context switch involves stopping a program miming on aprocessor and storing the values of the registers, program counter plusany other operating system specific data that may be necessary to themain memory. For example, to prevent a single process from monopolizinguse of a processor certain parallel processor programs perform a timertick at intervals ranging from about 60 ticks per second to about 100ticks per second. If the process running on the processor is notcompleted a context switch is performed to save the state of theprocessor and a new process (often the task scheduler or “kernel”) isswapped in. As used herein, the kernel refers to a central module of theoperating system for the parallel processor. The kernel is typically thepart of the operating system that loads first, and it remains in mainmemory. Typically, the kernel is responsible for memory management,process and task management.

Frequent context switches can be quite computationally intensive andtime consuming, particularly for processors that have a lot ofregisters. As used herein, a register refers to a special, high-speedstorage area within a processor. Typically, data must be represented ina register before it can be processed. For example, if two numbers areto be multiplied, both numbers must be in registers, and the result isalso placed in a register. The register may alternatively contain theaddress of a memory location where data is to be stored rather than theactual data itself. Registers are particularly advantageous in that theycan typically be accessed in a single cycle. Program compilers typicallymake use of as many software-configurable registers as are availablewhen compiling a program.

One prior art task management system used for cell processors is knownas SPU Threads. A “thread” generally refers to a part of a program thatcan execute independently of other parts. Operating systems that supportmultithreading enable programmers to design programs whose threadedparts can execute concurrently. SPU Threads operates by regarding theSPUs in a cell as processors for threads. A context switch may swap outthe contents of an SPU's local storage to the main memory and substitute256 kilobytes of data and/or code into the local storage from the mainmemory where the substitute data and code are processed by the SPU. Acontext switch is the computing process of storing and restoring thestate of a SPU or PPE (the context) such that multiple processes canshare a single resource. Context switches are usually computationallyintensive and much of the design of operating systems is to optimize theuse of context switches.

Unfortunately, interoperating with SPU Threads is not an option forhigh-performance applications. Applications based on SPU Threads havelarge bandwidth requirements and are processed from the PPU.Consequently SPU-threads based applications are not autonomous and tendto be slow. Because SPU Threads are managed from the PPU, SPU contextswitching (swapping out the current miming process on an SPU to anotherwaiting process) takes too long. Avoiding PPU involvement in SPUmanagement can lead to much better performance for certain applications

To overcome these problems a system referred to as SPU Runtime System(SPURS) was developed. In SPURS, the memory of each SPU has loaded intoit a kernel that performs scheduling of tasks handled by the SPU. Groupsof these tasks are referred to as Tasksets. SPURS is described in PCTApplication Publication number WO2007020739 to Keisuke Inoue and SeijiMurata entitled “SCHEDULING METHOD, AND SCHEDULING DEVICE”, and in USPatent Application Publication No. 20050188373, to Keisuke Inoue,Tatsuya Iwamoto and Masahiro Yasue entitled “METHOD AND APPARATUS FORTASK MANAGEMENT IN A MULTI-PROCESSOR SYSTEM”, and in US PatentApplication Publication No. 20050188372 to Keisuke Inoue and TatsuyaIwamoto entitled “METHOD AND APPARATUS FOR PROCESSOR TASK MIGRATION IN AMULTI-PROCESSOR SYSTEM” and in US Patent Application Publication No.20060190942 to Keisuke Inoue and Masahiro Yasue entitled “PROCESSOR TASKMIGRATION OVER A NETWORK IN A MULTI-PROCESSOR SYSTEM”, the disclosuresof all four of which are incorporated herein by reference.

In traditional Multi-Threading an operating system (OS) scheduler willpre-empt any running thread to make a scheduling decision. The OS maymanage any number of software threads that are configured byapplications. Processors have a fixed number of hardware threads that isoften fewer than the number of software threads in an application.Consequently, the hardware threads must be shared by the softwarethreads, e.g., by time-slicing or cooperative yielding. The number ofhardware threads determines how many software threads can runconcurrently. One hardware thread can only process one software threadat a time. The number of hardware threads is dependent on the type ofprocessor hardware involved. For example, the PPE of the Cell Processorhas two hardware threads. Co-processors and cache may be shared amongdifferent software threads. Each thread has associated with it acontext, which contains information relevant to the state of executionof the thread. Such information may include the values stored inregisters, a program counter value, and the like.

Current implementations of SPU Threads (also referred to as SPE Threads)for the Cell processor apply a traditional thread model to SPUs. Theoverhead for context switching these SPU Threads can be very highcompared to traditional threads because the entire SPE Local Store(currently 256 KB) and SPU registers (currently about 2 KB) must besaved and restored by the PPE. Certain SPU threads need to run on aregular basis, e.g., in certain Cell processor-based video gameapplications a high priority SPU thread runs every 16 ms. Currently, ifa group of ganged SPU threads are running on all available SPUs theentire gang must be swapped out even though the high priority SPU threadmay only require one SPU. The purpose of the ganged SPU Thread Group isso that the SPU Threads can safely communicate between each other's SPULocal Stores. If they were split up into individual SPU Threads,communication between SPU Local Stores is not supported by certainOperating Systems.

In addition to being detrimental to performance, context switches areoften unnecessary. For example, an application often has its contextresident in main memory. The space available in SPU Local Store isusually not big enough to store everything an SPU will need.Consequently some data and/or code may be stored in main memory where itis managed by the application. It is therefore often redundant for theoperating system to do the context switch.

It is within this context that embodiments of the present inventionarise.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 is a schematic diagram of a computer system according to anembodiment of the present invention.

FIG. 2 is a flow diagrams illustrating multi-threaded processingaccording to an embodiment of the present invention.

FIG. 3 is a block diagram of a computer system according to anembodiment of the present invention.

FIGS. 4A-4B are timing diagrams illustrating multi-threaded processingaccording to embodiments of the present invention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the exemplary embodiments of the invention described below are set forthwithout any loss of generality to, and without imposing limitationsupon, the claimed invention.

Embodiments of the present invention present modifications of standardmodels of multi-threaded computer operation such as the SPU Threadmodel. As used herein, the term multi-threaded or multi-threading meansthat the operating system (OS) is configured such that different partsof a single program (called threads) may run concurrently. Althoughembodiments of the present invention may be applied to anymulti-threaded computer system, the performance advantages may be mostapparent on a system with the following characteristics:

a) The processing system includes an asymmetric multi-core processor,e.g., a PPE and SPE(s), as in a Cell Processor.

b) One or more limited cores (e.g., SPE) is of a type that cannotperform a full thread context switch on itself.

c) The remaining core(s) (e.g., PPE) must perform context switches infull or in part on behalf of the limited cores.

d) A context switch of the limited core processor results in undesiredoverhead for the system.

According to embodiments of the present invention a computer systemhaving one or more processors coupled to a memory may implementmulti-threaded processing in a way that significantly reduces the needfor context switches or avoids them altogether. In particular anoperating system may notify a first thread miming on one or more of theprocessors of a pre-emption by a second thread having a higher prioritythan the first thread. Such notification is referred to herein aspre-emption notification. The operating system may also notify the firstthread of a time limit for pre-emption (referred to herein as apre-emption wait time attribute) associated with the second thread. Theapplication miming the first thread may yield the one the processor(s)held by the first thread to the second thread within the time limitwithout saving a context of the first thread if the first thread canwind up within the time limit. This process is referred to herein as ano-context-save yield. Alternatively, a context switch from the firstthread to the second thread may be performed if the first thread cannotwind up within the time limit.

As discussed above, in traditional multi-threading the OS Scheduler willpre-empt any running thread to make a scheduling decision. Inembodiments of the invention, by contrast, a computer system may beconfigured, e.g., through suitable programming, to provide moreinformation from an application about a thread to allow the operatingsystem (OS) to make a more cooperative scheduling decision. By way ofexample, the OS may notify an application that a certain low-prioritythread will be pre-empted unless the application gives up the hardwarethread within some specified amount of time. If the low-priority threadyields the hardware thread within the specified time, a new thread maybe loaded into the hardware thread. If the low-priority thread cannotyield the hardware thread within the specified time, a conventionalcontext switch may be performed.

Multi-threaded processing having the features described above may beimplemented on a computing system 100, as depicted in FIG. 1. Thecomputing system 100 may be a component of a device. Such devicesinclude, but are not limited to personal computers, laptop computers,hand held internet browsers and/or email devices, Voice over InternetProtocol (VoIP) phones, video game consoles, hand-held video gamedevices, portable email devices, portable internet browsers and thelike. In a preferred embodiment, the system 100 is a component of avideo game console. The system 100 may include processor module 101 anda memory 102 (e.g., RAM, DRAM, ROM, and the like). In some embodiments,the processing system 100 may have multiple cell processor modules 101.The processor module 101 generally includes a main processor elementsMPE and one or additional processor elements APE₁ . . . APE_(N). Theadditional processor elements may each include an associated localmemory. Each additional processor may run code and/or store dataassociated with one or more threads. By way of example, the additionalprocessor elements may be processors of the same type as the mainprocessor element. Such a configuration is sometimes referred to as amulti-core processor, e.g., a dual core or quad-core or processor.Alternatively, the additional processor elements may be co-processorelements or synergistic processor elements. The term co-processorgenerally refers to a special-purpose processing unit that assists acentral processing unit (CPU) in performing certain types of operations.Examples of coprocessors include math coprocessors that performmathematical computations, such as floating-point operations. Mathcoprocessors are sometimes also called numeric and floating-pointcoprocessors. In addition, coprocessors include graphics coprocessors(sometimes called accelerator boards) that are used to manipulategraphics images.

The system 100 may also include well-known support functions 110, suchas input/output (I/O) elements 111, power supplies (P/S) 112, a clock(CLK) 113 and cache 114. The system 100 may optionally include a massstorage device 115 such as a disk drive, CD-ROM drive, tape drive, orthe like to store programs and/or data. The controller may alsooptionally include a display unit 116 and user interface unit 118 tofacilitate interaction between the controller 100 and a user. Thedisplay unit 116 may be in the form of a cathode ray tube (CRT) or flatpanel screen that displays text, numerals, graphical symbols or images.The user interface 118 may include a keyboard, mouse, joystick, lightpen or other device. The cell processor module 101, memory 102 and othercomponents of the system 100 may exchange signals (e.g., codeinstructions and data) with each other via a system bus 120 as shown inFIG. 1.

As used herein, the term I/O generally refers to any program, operationor device that transfers data to or from the system 100 and to or from aperipheral device. Every transfer is an output from one device and aninput into another. Peripheral devices include input-only devices, suchas keyboards and mouses, output-only devices, such as printers as wellas devices such as a writable CD-ROM that can act as both an input andan output device. The term “peripheral device” includes externaldevices, such as a mouse, keyboard, printer, monitor, external Zip driveor scanner as well as internal devices, such as a CD-ROM drive, CD-Rdrive or internal modem or other peripheral such as a flash memoryreader/writer, hard drive.

The main processor MPE runs an operating system OS. The operating systemOS provides a software platform on top of which other programs, referredto herein as application programs or applications 103, may run. Theapplication program 103 may be implemented on the main processor elementMPE and additional processor elements APE₁ . . . APE_(N) as threadsincluding a main thread Thr_(M) running on the main processor elementMPE and additional threads Thr₁ . . . Thr_(N) running on the additionalprocessor elements APE₁ . . . APE_(N). The processor module 101 may alsoinclude one or more memory flow controllers MFC to facilitate transferof data between the processors and the main memory 102. The memory 102may include data and code configured as described below. Specifically,the memory may include an application program 103. Various processeswithin the application program 103 may be broken down into subtasks.Such subtasks may be implemented a work queue array 104, work queues 106and policy modules 108 each of which may include code, data or somecombination of both code and data. A detailed discussion of examples ofwork queues, work queue arrays and policy modules may be found incommonly-assigned US Patent Application Publication 2007 ______ (U.S.patent application Ser. No. 11/461,390), the entire contents of whichare incorporated herein by reference.

The flow diagram of FIG. 2 illustrates an example of method 200multi-threaded operation of a computer system like that shown in FIG. 1according an embodiment of the present invention. According to themethod a first thread may execute normally as indicated at 201. At 202the operating system may notify a first thread of a pre-emption by asecond thread having a higher priority than the first thread. By way ofexample, referring to FIG. 1 the first thread may be a thread running onone of the additional processor elements, e.g., thread Thr₁ running onadditional processor element APE₁. Alternatively, the first thread maybe running on the main processor element MPE. By way of example, thesecond thread may be part of the work queue 106. At 202 operating systemOS may also notify the first thread of a time limit for pre-emption(referred to herein as a pre-emption wait time attribute) associatedwith the second thread. By way of example, the wait time attribute T_(W)may be part of information about the thread stored, e.g., in the workqueue array 104. The application 103 may be configured yield the one theprocessor(s) held by the first thread to the second thread within thewait-time attribute without saving a context of the first thread if thefirst thread Thr₁ can wind up within the time limit. As indicated at 204the operating system OS may wait for the first thread to wind upexecution. The first thread may wind up execution by completing itsexecution within the allotted wait time T_(W). Alternatively, the firstthread may wind up execution by saving data that it needs to save beforeexiting. In addition, in some situations, the first thread may wind upexecution by removing a lock on a mutex 122 so that other threads mayaccess to the mutex 122 after the first thread exits. If the firstthread winds up its execution within the allotted wait time T_(W), thefirst thread may yield the processor to the second thread without acontext switch as indicated at 206. It is possible for the second threadto start earlier than its wait time, if the processor is yieldedearlier.

If the first thread does not wind up execution within the wait timeT_(W), the first thread may be preempted as indicated at 208. A contextswitch from the first thread to the second thread may then be performed,as indicated at 210. It is advantageous for the operating system OS toperform the context switch, because the threads involved may be fromseparate processes and may not be able to access each other's memoryareas. Subsequently, the second thread may be scheduled for execution onthe processor, as indicated at 212 and may execute as indicated at 214.

The processor module 101 may manage multi-threaded operation asdescribed above in response to data and program code instructions of theoperating system OS and application program 103 which may be stored andretrieved by the memory 102 and executed by the processor module 101.Code portions of the operating system OS and/or application program 103may conform to any one of a number of different programming languagessuch as Assembly, C++, JAVA or a number of other languages. Theprocessor module 101 forms a general-purpose computer that becomes aspecific purpose computer when executing programs such as the programcode 103. Although the program code 103 is described herein as beingimplemented in software and executed upon a general purpose computer,those skilled in the art will realize that the method of task managementcould alternatively be implemented using hardware such as an applicationspecific integrated circuit (ASIC) or other hardware circuitry. As such,it should be understood that embodiments of the invention can beimplemented, in whole or in part, in software, hardware or somecombination of both. In one embodiment, among others, the operatingsystem OS and/or program code 103 may include a set of processorreadable instructions that implement a method having features in commonwith the method 200 of FIG. 2, described above.

By way of example, and without loss of generality, the processor module101 may be a cell processor. FIG. 3 is a schematic depiction of a cellprocessor computer system 300 according to an embodiment of the presentinvention. In the embodiment depicted in FIG. 3, the cell processorsystem 300 may include a power processor unit (PPU) 302 and severalsynergistic processing elements (SPE) 304. In FIG. 3, a single PPU 302and eight SPE 304 are shown for the sake of example. Cell processorshaving two or more PPU and a greater or lesser number of SPU may be usedin conjunction with embodiments of the present invention.

The computing system 300 may be a component of a device. Such devicesinclude, but are not limited to personal computers, laptop computers,hand held internet browsers and/or email devices, Voice over InternetProtocol (VoIP) phones, video game consoles, hand-held video gamedevices, portable email devices, portable internet browsers and thelike. In a preferred embodiment, the system 300 is a component of avideo game console, such as the PlayStation3® from Sony ComputerEntertainment, Inc. of Tokyo, Japan.

The PPU 302 acts as a controller for the SPE 304, which handle most ofthe computational workload. The PPU 302 may also be used to runconventional operating systems if it is sufficiently similar to other64-bit PowerPC processors, and if the SPE 304 are designed forvectorized floating point code execution. The PPE may also include acache 307. By way of example, the cache 307 may contain a 32 kilobyteinstruction and data Level 1 cache and a 512 kilobyte Level 2 cache. Adetailed description of a particular example of a cell processorarchitecture known as Cell Broadband Engine Architecture (CBEA) isdescribed in detail, e.g., in Cell Broadband Engine Architecture,copyright International Business Machines Corporation, Sony ComputerEntertainment Incorporated, Toshiba Corporation Aug. 8, 2005 a copy ofwhich may be downloaded at http://cell.scei.co.jp/, the entire contentsof which are incorporated herein by reference.

The PPE 302 and SPE 304 can exchange code and data with each other overan exchange interface bus (EIB) 303. The PPE 302 and SPE 304 may alsoexchange code and data stored in a main memory 306, e.g., via an elementinterface bus (EIB) 303 and a memory flow controller (MFC) 308 such as adigital memory access (DMA) unit or the like. The EIB 303 may be acircular bus having two channels in opposite directions. The EIB 303 mayalso be connected to the cache 307, (e.g. the level 2 cache), the MFC308, and a system interface 305 such as a FlexIO for externalcommunications.

Each SPE 304 includes a processor, sometimes known as a synergisticprocessor unit (SPU) and a local memory 310. Code and data obtained fromthe main memory 306 can be loaded into the local memory 310 so that theSPE 304 can process tasks. Each SPE 304 may include registers 313 forinstructions and or data used during execution of tasks. The localmemory 310 of each SPE 304 may include a stack 311 to facilitateprocessing of tasks. As used herein and as is generally known, the term“stack” generally refers to a type of data structure in which items ofdata are removed in the reverse order from that in which they are added,so that the most recently added item is the first one removed. Such astructure is also sometimes known as a last-in, first-out (LIFO).

Each SPE 304 may have a number of features that are different fromconventional processors such as the PPE 302. For example, a conventionalprocessor allows code to address a large memory space, and transparentlycaches the memory to the processor's smaller cache memory. Unlikeconventional processors such as the PPE 302, an SPE 304 generally lacksa cache hierarchy. The lack of a cache hierarchy makes throughput to themain memory 306 much faster for an SPE 304 than for a conventionalprocessor. Access to the local memory 310 may be even faster as long asthe SPE 304 has been explicitly programmed to DMA data to or from thelocal memory 310.

In some embodiments, the local memory 310 of each SPE 304 may contain asoftware scheduler program 312. Preferably, the scheduler program 312takes up only a small fraction of the total memory space available ineach local memory 310 (e.g., less than about 1% of each SPE localmemory). The heart of the scheduler program 312, often referred to as a“Kernel”, may take up as little as about 2 KB resident on each SPE 304.For a 256K local memory, this represents about 0.8% SPE Local Storeusage.

In embodiments of the present invention, the PPE 302 may be programmedwith an operating system OS. The operating system OS provides a softwareplatform on top of which other programs, e.g., a application programAPP, may run. Embodiments of the present invention may take advantage ofthe fact that a program, such as the application APP often already has asignificant amount of its context data resident in main memory 306. Insuch cases it may therefore be redundant for the operating system OS todo a context switch. For example, in the particular case of the cellprocessor shown in FIG. 3, the local store 310 of a SPE 304 may not belarge enough to store everything that the corresponding SPU will need.Some code and/or data associated with the thread may be stored in mainmemory 306 where it can be managed by the application APP that isrunning the thread. For example, the main memory 306 may store threaddata 317 for a plurality of threads 317 ₁ . . . 317 _(N). The threaddata may include priorities P₁ . . . P_(N), executable linkable filesELF₁ . . . ELF_(N) and context data CTX₁ . . . CTX_(N) correspondinglyassociated with the threads 317 ₁ . . . 317 _(N).

In prior art multi-threading systems, if a higher priority thread iswaiting and ready, the OS will go through a context save and store. Thisis unacceptable on certain multi-processor systems, such as cellprocessors, where the SPE context switch is much more expensive than acontext switch of a typical multi-threaded processor such as PowerPC orX86. In embodiments of the present invention, by contrast, the threads317 ₁ . . . 317 _(N) may have associated with them correspondingwait-time attributes T_(W1) . . . T_(W2). Each wait-time attributeT_(W1) . . . T_(W2) specifies how long the corresponding thread can waitbefore pre-empting a lower priority thread.

In embodiments of the present invention, two or threads may be groupedtogether into a thread group. For example, as illustrated in FIG. 3,threads 317 ₁ . . . 317 _(N) may be grouped together in a first threadgroup 318 and threads 319 ₁, 319 ₂ . . . 319 _(N) may be groupedtogether in a second thread group 320. Each group may have a priorityand wait time attribute associated with it. The priorities and wait timeattributes for the threads within a group may be the same as each otheror may be different from each other depending on the nature of thethread group.

In some embodiments of the present invention, two or more threads may begrouped together in a thread group in such a way that they may bescheduled and/or unscheduled independently or all together. There areadvantages to being able to schedule and/or unscheduled threadsindependently. For example in SPURS, five different SPU threads aretypically ganged together as a single SPURS instance. In such a case,either all five SPU threads are miming on the SPUs or they are not. Ifsome high-priority thread needs to run on one SPU all five SPU threadsin the SPURS thread group must be swapped out so that the high-prioritythread can run. Not all thread groups need to work this way, however. Inthe context of embodiments of the invention, it is useful for theoperating system OS to be able to determine whether a thread group isganged or independent. Therefore, in embodiments of the invention threadgroups may be characterized by a thread group type. Specifically, thethread group type may identify whether the thread group isconstant-sized or variable-sized. In the case of a constant-sized threadgroup, the threads may be gang-scheduled as described above. In the caseof a variable-sized thread group, the number of threads in the group mayvary as determined by the OS.

Some SPU systems, such as SPURS, can execute application code withoutdepending on having a fixed number of SPU threads. In this case, avariable-sized SPU thread group may be used. Other systems may requirealways a specific number of SPUs. In this case, a constant-sized groupwould be used.

The variable- and constant-sized thread groups may have some upper limiton the number of threads in the group, e.g. up to 6 in the case ofcertain cell processor implementations. The upper limit on the number ofthreads in a group is generally determined by the number of availablelimited-core processors (e.g., SPUs in the case of a Cell Processor).The number of available SPUs may be less than the total number of SPUsphysically present in the Cell Processor, e.g., for reasons having to dowith yield and/or operating system configuration. In the particularexample of a cell processor, the operating system may runtime manage thenumber of SPU Threads assigned to variable Groups. Examples ofscheduling of SPU Thread groups, such as job streaming (also known asSPU Task Management (STM)), are described in detail, e.g., incommonly-assigned US Patent Application Publication 2007 ______ (U.S.patent application Ser. No. 11/461,390), the entire contents of whichare incorporated herein by reference. A new SPU Thread may be added to avariable Group, e.g., by starting a fresh copy of its ELF sections on aSPU or other processor. The default SPU Thread may be configured, e.g.,with parameters specified to the OS when the application created the SPUThread group.

Due to the independent, one-at-a-time, nature of thread scheduling in avariable-sized thread group, threads executing in a variable-sized Groupmay always yield with no context save.

Constant-sized groups may preferably yield with no-context-save, but mayfall back on a full context-save if they are preempted by the OS.However, as discussed above, the possibility exists that a thread maynot be able to exit within a given wait-time attribute. In such a case,the thread may be preempted. However, if a thread in a variable-sizedgroup is preempted, its context must be saved and restored. It istherefore useful to have a thread Group attribute for a preemptionpolicy implemented by the OS. Specifically, the thread group attributemay specify whether threads in a given group are to be a) preempted as agroup, or b) preempted individually. In the example depicted in FIG. 3,threads to be preempted as a group are indicated by type=1 and threadsto be preempted individually are indicated by type=2. Specifically,threads 317 _(k) . . . 317 _(N) in thread group 318 are to be preemptedas a group and threads 319 _(k) . . . 319 _(N) in the second threadgroup 320 are to be preempted individually. It is noted that althoughthe concept of a thread group type is discussed herein in the context ofa Cell processor example, those of skill in the art will recognize thatthe concept may be applied to any type of multi-threaded computerprocessing.

If a thread is preempted individually, other threads within its groupmay communicate with each other safely. The ability to preempt threadsindividually may be useful, e.g., where there is no a priori knowledgeabout competing threads. The ability to preempt threads individually maybe potentially useful for unknown Cell processor applications having twoor more independent processes with their own internal thread usages andeach process involves one or more SPEs. In embodiments of the invention,it is often desirable to carefully choose the point at which a givenpreempted thread drops out. For example, a thread may drop out when thescheduler 312, e.g., a SPURS or SPMM kernel, is executing if there is nointer SPE communication taking place at that point. Alternatively, theapplication APP may determine safe point for a preempted thread to dropout.

Operation of the system 300 in accordance with embodiments of thepresent invention may be understood by simultaneously referring to FIG.3 and FIGS. 4A-4B. By way of example, the first thread group 318 may beexecuting on an SPE 304 as indicated at 402 in FIG. 4A and FIG. 4B. Thesecond thread group 320 may become available while the first threadgroup 318 is running on the SPE 304. For the sake of example, supposethat the first thread group 318 has a wait time attribute T_(W1)=1000microseconds, a priority P₁=10 and a thread group type=1 and that thesecond thread group 320 has a wait time attribute T_(W2)=50microseconds, a priority P₂=1 and a thread group type=2. In this case,the first thread group 318 has a lower priority P₁ than the priority P₂of a second thread group 320. When the second thread group 320 becomesready for execution, the operating system OS may signal the applicationAPP and start a timer set according to the 50-microsecond wait-timeattribute T_(W2) of the high-priority second thread group 320.

As shown in FIG. 4A, The operating system OS may notify the SPE 304running the first thread group 318 (or whatever process is running thefirst thread group 318) by sending a preemption notification 403 thatwarns the SPE 304 of a pending preemption of first thread 318 (or itsparent process) by the second thread group 320. The operating system OSmay also notify the SPE 304 of the allotted wait time T_(W2) in which toyield. By way of example, the operating system OS may call a call-backassociated with the running lower priority thread. In the particularcase of a cell processor, the call-back may be implemented as a functioncall on the PPU process that owns the first thread group 318. Thefunction call may include the wait time attribute T_(W2) as a parameter.The aforementioned PPU process may be the application APP or a processwithin the application. The first thread group 318 or application APPmay be configured so that upon receipt of the preemption notification403, the first thread 318 thread may complete execution, save data thatneeds to be saved or take other appropriate action in response to yieldinstructions as indicated at 404, and then exit at 406 withoutperforming a context save. As discussed above, this procedure isreferred to herein as a no-context-save yield. The no-context save yieldmay include removing a spin lock on a mutex 322 prior to exiting.Alternatively, the instructions 404 implementing the yield may includeone or more instructions that, when executed, maintain the first threadin a ready state until a subsequent call to execute the first thread. Itis possible for the second thread group 320 to start before its50-microsecond wait time has elapsed, if the SPE 304 yields earlier. Thefirst thread group 318 (or application APP) may notify the OS that ithas exited as indicated at 405, whereupon the operating system OS maydeposit the second thread group 320 into the SPE 304 for execution. Thesecond thread may then execute as indicated at 408. Specifically, theoperating system OS may notify the PPU 302 of the yield and the secondthread group 320 may be immediately scheduled for execution.

The no-context-save yield instructions 404 may be configured so that thenext time the first thread group 318 is to be swapped into the SPE 304the application APP can load a fresh copy of the executable linkablefiles ELF₁ . . . ELF_(N) for the threads 317 ₁ . . . 317 _(N) in thefirst thread group 318 re-set to initialized values. In someembodiments, the no-context-save yield may be implemented at theoperating system level, e.g., by an instruction that tells the OS todestroy the thread. When the destroyed thread is to be resumed, theapplication has to create the same thread. Although exits that destroy athread have been implemented at the application level, it is believed tobe novel to implement a no-context save yield at the OS level. Inaddition, an OS that supports a no-context-save yield natively isbelieved to be counterintuitive, because when the thread is to beresumed by the OS, the OS starts a fresh instance of the thread in thesame way as when the thread was started for the first time. Intuitively,resuming a thread implies that there was some OS-maintained context tobring up so that the thread can continue where it left off. One of skillin the art would therefore consider it impossible to resume a thread inthe absence of a context save. In embodiments of the present invention,by contrast, the application determines where it left off each time thethread is resumed.

In some circumstances, the first thread group may be unable to yield theSPE 304 within the allotted wait-time attribute T_(W2). For example, asillustrated in FIG. 4B, if the SPE 304 does not yield within the 50-μswait-time T_(W2), the operating system OS may send a preemption command407 to the SPE 304. The first thread group 318 is preempted, and contextsave 410 is then performed. The context save 410 may involve savingregister values and program counter values and the like to main memory306. The second thread group 320 may then execute as indicated at 412.

The wait-time attribute for a given thread may be determinedempirically. In some embodiments, it may be desirable that the wait-timeattribute be less than an amount of time required for a context switch.In embodiments of the present invention, the wait-time attributes forcooperative threads T_(W1) . . . T_(WN) running concurrently may bejudiciously chosen so that higher-priority threads can be brought up andrunning significantly faster than would otherwise be possible usingcontext switches. In fact, if the threads are sufficiently cooperativeand the wait-time attributes T_(W1) . . . T_(WN) are properly chosen,the application APP may be configured in such a way as to avoid contextswitches entirely. It is noted that a default wait time attribute valueof 0 for a given thread results in immediate preemption with contextsaves, which is the current behavior.

Advantages associated with embodiments of the invention may be seen uponconsidering the following example. As discussed above, certainmulti-threaded applications may include a repeating high priority threadthat is run at regular intervals of time. In prior art multi-threadedprocessing methods and systems, if this thread preempts a gang-scheduledthread group, all the threads in the group need to be swapped out evenif the repeating thread needs only a single SPE. If all thread groupsare gang-scheduled then there will be frequent interruptions inprocessing and a significant reduction in performance as the repeatingthread regularly preempts gang-scheduled thread groups. The workaroundsto avoid this are unsatisfactory. One workaround is to split up thethread group into one large group and one small group, so that the smallgroup will be preempted by the periodic high priority thread.Unfortunately, this results in the application managing two separateworkloads for each thread group. Another workaround is to avoid usingone of the SPEs entirely except for the high-priority thread, so thatthe high priority thread has a dedicated SPE to use. Unfortunately, thismakes the dedicated SPE unavailable for other work that it couldotherwise be doing. If, however, threads may be individually swapped outas defined by the variable-sized thread group, the repeating threadgroup may preempt a few individual threads, thereby allowing otherrelated threads to continue executing and freeing up valuable processorresources.

According to embodiments of the present invention, systems and softwareimplementing the features described herein may be madebackwards-compatible with prior software versions. For example, in someCell processor implementations, the interface between the operatingsystem and the application is a set of software instructions referred toas Syscalls. Currently, an SPU thread is defined by some data structureand the thread is initiated through a corresponding Syscall instructionto that data structure. The data structure may include attributes of thethread, pointers to code and/or data associated with the thread andother information. Embodiments of the present invention may beimplemented using similar data structures configured to accommodateadditional data, such as the wait-time attribute and thread group type.In embodiments of the invention new Syscalls may be created for suchdata structures. However, the new data structures may be configured suchthat they will also work with the existing Syscalls. Similarly, the newSyscalls may be configured to work with the old data structures. In thismanner, the new operating system may be made backwards compatible witholder applications.

While the above is a complete description of the preferred embodiment ofthe present invention, it is possible to use various alternatives,modifications and equivalents. Therefore, the scope of the presentinvention should be determined not with reference to the abovedescription but should, instead, be determined with reference to theappended claims, along with their full scope of equivalents. Any featuredescribed herein, whether preferred or not, may be combined with anyother feature described herein, whether preferred or not. In the claimsthat follow, the indefinite article “A”, or “An” refers to a quantity ofone or more of the item following the article, except where expresslystated otherwise. The appended claims are not to be interpreted asincluding means-plus-function limitations, unless such a limitation isexplicitly recited in a given claim using the phrase “means for.”

What is claimed is:
 1. In a computer system having one or moreprocessors coupled to a memory, a method for multi-threaded processing,comprising: notifying one or more of the processors miming a firstthread or thread group of a pre-emption by a second thread or threadgroup characterized by a higher priority than the first thread;notifying the one or more of the processors of a time limit forpre-emption associated with the second thread or thread group; andyielding the one or more of the processors to the second thread orthread group within the time limit without saving a context of the firstthread or thread group if the first thread or thread group can wind upwithin the time limit, or performing a context switch from the firstthread or thread group to the second thread or thread group if the firstthread or thread group cannot wind up within the time limit.
 2. Anon-transitory computer-readable medium having a set of computerreadable instructions embodied therein, the computer-readableinstructions being configured to implement a method for a multi-threadedprocessing on a computer system having one or more processors coupled toa memory, the computer-readable instructions comprising: one or morepre-emption-notification instructions that, when executed, notify one ormore of the processors miming a first thread or thread group of apre-emption by a second thread or thread group characterized by a higherpriority than the first thread or thread group; one or morewait-time-notification instructions that, when executed, notify thefirst thread or thread group of a time limit for pre-emption associatedwith the second thread or thread group; and one or more yieldinstructions that, when executed, yield the one or more of theprocessors to the second thread or thread group within the time limitwithout saving a context of the first thread or thread group if theprocessor can yield the first thread or thread group within the timelimit or perform a context switch from the first thread or thread groupto the second thread or thread group if the processor cannot yield thefirst thread or thread group within the time limit.
 3. Thecomputer-readable medium of claim 2 wherein the pre-emption-notificationinstructions are part of an operating system for the computer system. 4.The computer-readable medium of claim 2 wherein thewait-time-notification instructions are part of an operating system forthe computer system.
 5. The computer-readable medium of claim 2 whereinthe yield instructions include one or more instructions that are part ofan operating system for the computer system.
 6. The computer-readablemedium of claim 2 wherein the yield instructions include one or moreinstructions that, when executed, unlock a mutex previously locked bythe first thread or thread group before yielding the one or more of theprocessors to the second thread or thread group.
 7. Thecomputer-readable medium of claim 2 wherein the yield instructionsinclude one or more instructions that, when executed, maintain the firstthread in a ready state until a subsequent call to execute the firstthread or thread group.
 8. The computer-readable medium of claim 2wherein the time limit is less than a time needed for a context switchfrom the first thread or thread group to the second thread or threadgroup.
 9. The computer-readable medium of claim 2, wherein thecomputer-readable instructions further comprise one or more instructionsthat, when executed, group the first thread with one or more otherthreads in a group characterized by a thread group type, wherein thethread group type is either a constant-sized group of two or morethreads that are gang-scheduled or a variable-sized group of two or morethreads that can be scheduled and/or unscheduled one at a time.
 10. Thecomputer-readable medium of claim 9 wherein, if the thread group type isvariable-sized, the yield instructions are configured to yield only thefirst thread and no other threads in the group if only the first threadis pre-empted.
 11. The computer-readable medium of claim 9 wherein, ifthe thread group type is constant- sized, the yield instructions areconfigured to yield the first thread and all other threads in the groupeven if only the first thread is pre-empted.
 12. The computer-readablemedium of claim 2 wherein the context switch includes saving contextdata associated with the first thread in the memory.
 13. Thecomputer-readable medium of claim 2 wherein the yield instructionsinclude one or more instructions that, when executed, schedule thesecond thread for execution on the one or more of the processors.
 14. Acomputer system, comprising: one or more processors; and a memorycoupled to the one or more processors, wherein the computer system isprogrammed with a set of computer-readable instructions configured toimplement a method for a multi-threaded processing, thecomputer-readable instructions comprising: one or morepre-emption-notification instructions that, when executed, notify one ormore of the processors miming a first thread or thread group of apre-emption by a second thread or thread group characterized by a higherpriority than the first thread or thread group; one or morewait-time-notification instructions that, when executed, notify the oneor more of the processors of a time limit for pre-emption associatedwith the second thread or thread group; and one or more yieldinstructions that, when executed, yield the one or more of theprocessors to the second thread or thread group within the time limitwithout saving a context of the first thread or thread group if the oneor more of the processors can yield the first thread or thread groupwithin the time limit or perform a context switch from the first threador thread group to the second thread or thread group if the one or moreof the processors cannot yield within the time limit.
 15. The computersystem of claim 14 wherein the one or more processors include two ormore processors.
 16. The computer system of claim 15 wherein the two ormore processors include a power processor element and one or moresynergistic processor elements.
 17. The computer system of claim 14wherein the pre-emption-notification instructions are part of anoperating system for the computer system.
 18. The computer system ofclaim 14 wherein the wait-time-notification instructions are part of anoperating system for the computer system.
 19. The computer system ofclaim 14 wherein the yield instructions include one or more instructionsthat are part of an operating system for the computer system.
 20. Thecomputer system of claim 14 wherein the yield instructions include oneor more instructions that, when executed, unlock a mutex previouslylocked by the first thread or thread group before yielding the one ormore of the processors to the second thread.
 21. The computer system ofclaim 14 wherein the yield instructions include one or more instructionsthat, when executed, maintain the first thread or thread group in aready state until a subsequent call to execute the first thread.
 22. Thecomputer system of claim 14 wherein the time limit is less than a timeneeded for a context switch from the first thread or thread group to thesecond thread or thread group.
 23. The computer system of claim 14,wherein the computer-readable instructions further comprise one or moreinstructions that, when executed, group the first thread with one ormore other threads in a group characterized by a thread group type,wherein the thread group type is either a constant-sized group of two ormore threads that are gang-scheduled or a variable-sized group of two ormore threads that can be scheduled and/or unscheduled one at a time. 24.The computer system of claim 23 wherein, if the thread group type isvariable-sized, the yield instructions are configured to yield only thefirst thread and no other threads in the group if only the first threadis pre-empted.
 25. The computer system of claim 23 wherein, if thethread group type is constant-sized, the yield instructions areconfigured to yield the first thread and all other threads in the groupeven if only the first thread is pre-empted.
 26. The computer system ofclaim 14 wherein the context switch includes saving context dataassociated with the first thread or thread group in the memory.
 27. Thecomputer system of claim 14 wherein the yield instructions include oneor more instructions that, when executed, schedule the second thread orthread group for execution on the one or more of the processors.
 28. Thecomputer system of claim 14 wherein the one or more processors include apower processing element and one or more synergistic processingelements.
 29. The computer system of claim 14 wherein the one or moreprocessors include a multi-core processor.
 30. The computer system ofclaim 14 wherein the system is a component of a personal computer,laptop computer, hand held internet browser device, hand-held emaildevice, Voice over Internet Protocol (VoIP) phones, video game console,or hand-held video game device.
 31. The computer system of claim 30wherein the one or more processors include a power processing elementand one or more synergistic processing elements and the device is avideo game console.
 32. The computer system of claim 14 wherein the oneor more processors include an asymmetric multi-core processor.
 33. Thecomputer system of claim 32 wherein the multi-core processor includesone or more limited cores of a type that cannot perform a full threadcontext switch on itself.
 34. The computer system of claim 33 wherein,if a context switch is required for one or more of the limited cores,one or more remaining cores of the multi-core processor must performcontext switches in full or in part on behalf of the one or more limitedcores requiring the context switch.
 35. The computer system of claim 34wherein a context switch of the limited core processor results inundesired overhead for the computer system.